Quantity versus quality: A new approach to examine the relationship between technology use and student outcomes

The author argues that to examine the relationship between technology use and student outcomes, the quality of technology use—how, and what, technology is used—is a more significant factor than the quantity of technology use—how much technology is used. This argument was exemplified by an empirical study that used both angles to examine the association between technology use and student outcomes. When only the quantity of technology use was examined, no significant association was observed. However, when the quality of technology was examined by investigating the specific types of technology uses, a significant association was identified between technology use and all student outcomes. Furthermore, different types of technology use showed different influences on specific student outcomes. General technology uses were positively associated with student technology proficiency, while subject-specific technology uses were negatively associated with student technology proficiency. Social-communication technology uses were significantly positively associated with developmental outcomes such as self-esteem and positive attitude towards school. Entertainment/exploration technology use showed significant positive association with student learning habits. None of these technology uses had significant influence on student academic outcome. Specific suggestions for integrating technology into schools and future research were provided. Introduction In the last two decades, generous investments have been made in educational technology around the world. For example, the US had invested more than $66 billion in school technology in just 10 years (Quality Education Data, 2004). By 2004, China had spent 100 billion Yuan (about $13.2 billion) on educational technology (Zhao, 2005), and British Journal of Educational Technology Vol 41 No 3 2010 455–472 doi:10.1111/j.1467-8535.2009.00961.x © 2009 The Author. Journal compilation © 2009 Becta. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA. the annual expense on educational technology was projected to reach 35.5 billion Yuan in 2007 (Okokok Report, 2004). Ireland’s second national educational technology plan proposed to invest 107.92 million pounds in educational technology (Ireland Ministry of Education and Science, 2001). The generous investments were supported by the strongly held premise that technology can help students learn more efficiently and effectively, and as a result increase student academic achievement. This belief in the connection between technology and student achievement is a theme commonly emphasised in mission statements of educational technology projects and arguments to support educational technology investment (Zhao & Conway, 2001). For example, the first US national educational technology plan claims, ‘Properly used, technology increases students’ learning opportunities, motivation, and achievement’ (U.S. Department of Education, 1996, p. 10). The second plan assures that technology would ‘enhance learning and improve student achievement for all students’ (U.S. Department of Education, 2000, p. 4). The third plan further states that with new technologies, ‘10 years from now we could be looking at the greatest leap forward in achievement in the history of education. By any measure, the improvements will be dramatic’ (U.S. Department of Education, p. 11). However, this premise on the crucial role of technology in student achievement has not been substantially supported by empirical evidence. In fact, findings from different empirical studies focusing on the effect of technology on learning have been inconsistent and contradictory. On the one hand, some studies have identified significant positive impact of technology use on student outcomes in academic areas such as literacy development (Blasewitz & Taylor, 1999; Tracey & Young, 2006), reading comprehension and vocabulary (Scrase, 1998; Stone, 1996; Woehler, 1994), writing (Nix, 1998), mathematics (Elliott & Hall, 1997; Mac Iver, Balfanz & Plank, 1999) and science (Harmer & Cates, 2007; Lazarowitz & Huppert, 1993; Liu, Hsieh, Cho & Schallert, 2006; Reid-Griffin, 2003). For example, Tienken and Wilson (2007) compared seventhgrade students whose teachers used mathematics websites and presentation software in their classrooms with students whose teachers did not teach with these technology tools. They found that the use of these technology tools had a positive effect on students’ learning of basic mathematic skills. In addition, positive impacts have been identified in student developmental areas, including attitude towards learning and self-esteem (Nguyen, Hsieh & Allen, 2006; Sivin-Kachala & Bialo, 2000), motivation, attendance and discipline (eg, Matthew, 1997). For example, using a mixed-method design, Wighting (2006) reported that using computers in the classroom positively affected students’ sense of learning in a community. Similarly, in the UK, a series of research studies have been conducted to examine the effect of the large-scale Tablet PC programmes, and findings reveal that the use of technology has improved student access to curriculum, communication and motivation (eg, Sheehy et al, 2005; Twining et al, 2006). On the other hand, several other researchers have come to very different conclusions. Some argue that technology use may not have any positive impact on student outcomes. For example, The Organization for Economic Co-operation and Development’s (OECD) Programme for International Student Assessment 2003 study found that stu456 British Journal of Educational Technology Vol 41 No 3 2010 © 2009 The Author. Journal compilation © 2009 Becta. dents using computers most frequently at school did not necessarily perform better than students using technology less frequently, and the impact of technology use on student math achievement varied by countries (OECD, 2005). In March 2007, the Institute of Education Sciences released an influential report titled Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort. This study, intended to assess the effects of 16 computer software products designed to teach firstand fourthgrade reading and sixth-grade math, using a rigorous random assignment design, found that ‘test scores in treatment classrooms that were randomly assigned to use products did not differ from test scores in control classrooms by statistically significant margins’ (Dynarski et al, 2007, p. xiii). Furthermore, some studies suggest that technology use might even harm children and their learning (eg, Healy, 1998; Stoll, 1999). For example, Waight and Abd-El-Khalick (2007) found that the use of computer technology restricted rather than promoted ‘inquiry’ in a sixth-grade science classroom. Mixed findings have also emerged from large-scale international studies. A study of the Trends in International Mathematics and Science Study (TIMSS) reported that technology use was negatively related to science achievement among eighth graders in Turkey (Aypay, Erdogan & Sozer, 2007). Another TIMSS study found that while medium use of computer technology was related to higher science scores, extensive use was related to lower science scores (Antonijevic, 2007). Similarly, based on data collected from 175 000 15-year-old students in 31 countries, researchers at the University of Munich announced that performance in math and reading had suffered significantly among students who had more than one computer at home (MacDonald, 2004). Schacter (1999) also identified some negative impacts on student achievement through the review of five large-scale studies that employed diverse research methods to examine the impact of educational technology. Discouraged by these findings, some people have come to the conclusion that putting computers in classrooms has been wasteful and pointless (Oppenheimer, 2003). Existing research on the relationship technology on student learning presents a mixed message (Andrews et al, 2007; O’Dwyer, Russell, Bebell & Tucker-Seeley, 2005; Torgerson & Zhu, 2003). Such mixed and often conflicting findings make it difficult to draw conclusions about the effects of technology, to provide meaningful advice to those who make decisions about technology investment in education and to make practical suggestions for integrating technology into schools. There are at least two problems contributing to the controversy over the relationship between technology use and student outcomes. The first is that technology is often examined at a very general level (Zhao, 2003). Many studies ‘treat technology as an undifferentiated characteristic of schools and classrooms. No distinction is made between different types of technology programs’ (Wenglinsky, 1998, p. 3). We know that technology is a very broad term that includes many kinds of hardware and software. These technologies may have different impacts on student outcomes. Even the same technology can be used differently in various contexts to solve all kinds of problems (Zhao) and thus have ‘different meanings in different settings’ (Peyton & Bruce, Quantity versus quality on technology use 457 © 2009 The Author. Journal compilation © 2009 Becta. 1993, p. 10). Treating technology as if it is a single thing obscures the unique characteristics of different technologies and their uses. The second problem is the focus of the studies. Most studies focus on the impact of the quantity of technology use, in other words, how much or how frequently technology is used, but ignore the quality of technology use, that is, how technology is used. For example, many studies examine the relationship between how much time students spend on using computers or how often they use computers and their achievement (eg, Du, Havard, Yu & Adams, 2004; Mann, Shakeshaft, Becker & Kottkamp, 1999). However, research suggests that the quality of technology use is more critical to student outcomes than the quantity (Burbules & Callister, 2000; Lei & Zhao, 2007; McFarlane, 1997). As Goldenberg (2000) pointed out that ‘what really matters is not the use of technology, but how it is used’ (p. 2). Thus, the necessary next step is to examine how different uses of diverse technologies affect student learning. This study investigates the relationship between technology use and student outcomes by comparing the association between the quantity of technology and student outcomes with the association between the quality of technology use and student outcomes. This approach differs from many previous studies in at least two aspects. First, it studies technology at a more specific level instead of technology in general. Second, to better discern the quality of technology use, this study focuses on different uses of technology rather than on specific technological objects such as hardware or software. ‘Technology use’ is the application of a technology function to solve practical problems (Zhao, 2003). The focus is technology-in-context. Examining technologies from this angle allows us to discern the different uses of the same technologies so that the nature of different technology uses can be better understood. Analytical framework